Skip to content

[release-5.0] NO-ISSUE: Increase greenboot and robot timeouts for release scenarios - #7318

Closed
agullon wants to merge 2 commits into
openshift:release-5.0from
agullon:NO-ISSUE-release-timeouts-release-5.0
Closed

[release-5.0] NO-ISSUE: Increase greenboot and robot timeouts for release scenarios#7318
agullon wants to merge 2 commits into
openshift:release-5.0from
agullon:NO-ISSUE-release-timeouts-release-5.0

Conversation

@agullon

@agullon agullon commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Increase greenboot healthcheck timeout from 600s (10 min) to 1200s (20 min) for release scenarios
  • Increase Robot Framework test execution timeout from the CI-configured 45m to 60m for release scenarios
  • Both changes are scoped to release scenarios only via ci_phase_boot_and_test.sh

Context

Release scenarios running upgrade paths with LVMS workloads followed by full standard suites
were hitting timeout limits under I/O contention on x86 (c5.metal, 4750 Mbps EBS bandwidth)
when many VMs boot and pull container images from the mirror registry in parallel. The ARM
instance (m7g.metal, 20000 Mbps EBS) has ~4.2x more I/O bandwidth and doesn't hit these limits.

Specific failures observed in PR #7297:

  • el96-y2@el98-lrel@lvms-standard1/2: greenboot healthcheck failed — pods not ready within 600s
  • el98-y1@el98-lrel@lvms-standard: Robot Framework killed by SIGTERM at 45m (44/52 tests had passed)

These timeouts are ceilings, not floors — greenboot polls and exits immediately when pods are ready,
and robot finishes whenever tests complete. The happy-path duration is unaffected.

The el10 lvms-standard scenarios and the optional scenario already set these values per-scenario.
This change centralizes them for all release scenarios in ci_phase_boot_and_test.sh.

Test plan

  • Verify release CI jobs pass with the new timeouts
  • Confirm non-release scenarios (presubmits, periodics, c2cc) are unaffected

🤖 Generated with Claude Code

@agullon
fix: increase greenboot and robot timeouts for release scenarios
3f4d707 
Release scenarios running upgrade paths with LVMS workloads followed
by full standard suites were hitting timeout limits under I/O contention
on x86 (c5.metal, 4750 Mbps EBS) when many VMs boot and pull images in
parallel. Increase greenboot healthcheck timeout from 600s to 1200s and
robot framework timeout from 30m (CI-overridden to 45m) to 60m for
release scenarios only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
@openshift-merge-bot

openshift-merge-bot Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@coderabbitai

coderabbitai Bot commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e7d239cf-2d8f-4252-a4d3-8637122bd2f7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@agullon agullon mentioned this pull request Sep 3, 2026
Closed
2 tasks
@openshift-ci

openshift-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: agullon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 3, 2026
@agullon agullon mentioned this pull request Sep 3, 2026
Merged
1 task
@agullon
chore: remove per-scenario timeout overrides now centralized
a2b7743 
GREENBOOT_TIMEOUT=1200 and TEST_EXECUTION_TIMEOUT=60m are now set
centrally in ci_phase_boot_and_test.sh for all release scenarios.
Remove the redundant per-scenario overrides.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED
@agullon agullon changed the title Increase greenboot and robot timeouts for release scenarios NO-ISSUE: Increase greenboot and robot timeouts for release scenarios Sep 3, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 3, 2026
@openshift-ci-robot

openshift-ci-robot commented Sep 3, 2026

Copy link
Copy Markdown

@agullon: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

  • Increase greenboot healthcheck timeout from 600s (10 min) to 1200s (20 min) for release scenarios
  • Increase Robot Framework test execution timeout from the CI-configured 45m to 60m for release scenarios
  • Both changes are scoped to release scenarios only via ci_phase_boot_and_test.sh

Context

Release scenarios running upgrade paths with LVMS workloads followed by full standard suites
were hitting timeout limits under I/O contention on x86 (c5.metal, 4750 Mbps EBS bandwidth)
when many VMs boot and pull container images from the mirror registry in parallel. The ARM
instance (m7g.metal, 20000 Mbps EBS) has ~4.2x more I/O bandwidth and doesn't hit these limits.

Specific failures observed in PR #7297:

  • el96-y2@el98-lrel@lvms-standard1/2: greenboot healthcheck failed — pods not ready within 600s
  • el98-y1@el98-lrel@lvms-standard: Robot Framework killed by SIGTERM at 45m (44/52 tests had passed)

These timeouts are ceilings, not floors — greenboot polls and exits immediately when pods are ready,
and robot finishes whenever tests complete. The happy-path duration is unaffected.

The el10 lvms-standard scenarios and the optional scenario already set these values per-scenario.
This change centralizes them for all release scenarios in ci_phase_boot_and_test.sh.

Test plan

  • Verify release CI jobs pass with the new timeouts
  • Confirm non-release scenarios (presubmits, periodics, c2cc) are unaffected

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@agullon agullon changed the title NO-ISSUE: Increase greenboot and robot timeouts for release scenarios [release-5.0] NO-ISSUE: Increase greenboot and robot timeouts for release scenarios Sep 3, 2026
@openshift-ci

openshift-ci Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

@agullon: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@agullon agullon mentioned this pull request Sep 4, 2026
Merged
@agullon

agullon commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Consolidated into #7327

@agullon agullon closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggiguash ggiguash Awaiting requested review from ggiguash
@kasturinarra kasturinarra Awaiting requested review from kasturinarra

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@agullon @openshift-ci-robot